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ABSTRACT 


Two alternative methods for optimizing an unconstrained non-1inear 
function are investigated and compared. The investigations are made 
subject to a restriction as to the number of function evaluations 
available to conduct the optimization procedures. Powell's method of 
conjugate directions is employed as the direct search method and is 
considered the reference method. The alternate method is based on 
fitting a quadratic surface to the available function evaluations and 
optimizing over the resulting fitted surface. The test functions 


considered in the investigation were limited to unimodal functions. 
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I. NATURE OF THE PROBLEM 


A. INTRODUCTION 

In Operations Research a major problem concerns the determination of 
the optimum response for an objective function. In many cases, the 
objective function may not lend itself to expression in closed analytical 
form. Information concerning the nature of the objective ‘function may 
only be available through computer simulation, field testing or a combina- 
tion of both. Furthermore, restrictions may exist with respect to the 
quantity of information available or obtainable due to the costs involved 
in securing the data. Thus, it is incumbent that the most efficient use 
be made of the limited information available in order to achieve the best 
possible results in determining the optimum response. 

There exist several search techniques that could be adapted to solve 
the problem as stated above. Generally, these search techniques could 
be classified as direct search techniques or gradient search techniques. 
Direct search methods do not require the computation of partial deriva- 
tives of the objective function. Determination of the optimum response 
is based solely on values of the objective function itself. On the other 
hand, gradient search methods compute values of partial derivatives of 
the objective function. These resulting values are then used in select- 
ing future search directions. Search techniques employing gradient 
methods require more information (i.e., additional function evaluations, 
computer simulations, etc.) than direct search procedures. Thus, with 
the presence of restrictions on the quantity of information available it 


would be more advantageous to use a direct search method. 


ad 





Direct Search methods can be broadly divided into three classifica- 
tions: tabulation methods, sequential methods and linear methods. 
Tabulation methods, as the name implies, result in a tabular listing of 
objective function responses for different values of the input variables. 
The "optimum" response is considered to be the "best" value found in the 
table. In the case of minimization problems, the "best" value would be 
the smallest value. An example of a tabulation method is random search. 

Sequential methods employ the use of geometric designs in the input 
variable space. The objective function is evaluated at the vertices of 
the geometric design. The sequence is repeated until the desired accuracy 
is achieved or the restriction of the quantity of information obtainable 
precludes additional testing. Factorial designs and the simplex methad 
are examples of sequential methods. 

Linear methods employ the use of a set of direction vectors through- 
out the conduct of the search. The objective function is evaluated at 
different points along these search directions and conduct of the search 
1s dependent upon the results obtained. The set of direction vectors 
may or may not be changed during the course of the search. Examples of 
linear methods are the alternating variable search and Powell's method 
of conjugate directions. 

An alternative approach to employing search techniques to solve the 
problem would be to use curve-fitting techniques. Using this technique, 
test points would be established in the input variable space and the 
objective function would be evaluated at these test points. A least- 
Square polynomial surface would then be fitted to these resulting 
objective function values. To complete the procedure the location of 


the "optimum" response of the fitted surface would be determined and the 





resulting input variable values would be input to the true objective 


function to determine the "optimum" response. 


B. STATEMENT OF THE PROBLEM 

The specific problem that was the subject of investigation was to 
compare the relative efficiency of a curve-fitting technique as opposed 
to a search technique. The assumption was made that a restriction exists 
on the quantity of information available (i.e., there were a finite number 
of functional evaluation that could have been made). A direct result of 
the above assumption was that the investigation was limited to those 
techniques which do not require the computation of partial derivatives. 
Furthermore, the investigation was restricted to test objective functions 
of a unimodal nature insuring the existence of only one true optimum 


response. 





Bie aA en MENTAL PROCEDURES 


A. SEARCH PROCEDURE 

Powell's method of conjugate directions was chosen as the direct 
Search technique to be used. Powell's method is considered to be one 
of the most efficient direct search techniques available [Ref.1]. Its 
efficiency is exceptionally good for functions that can be approximated 
by a quadratic in the region of the optimum. 

Powell's method commences with a search along a set of linearly 
independent directions that span the input variable space starting from 
an arbitrarily chosen initial point. The initial set of directions 
chosen are the co-ordinate directions. At the completion of each itera- 
tion a new direction is.defined and replaces one of the presently exist- 
ing directions only if the new set of directions is at least as efficient 
as the present set. If this is not the case, an additional iteration 1s 
conducted retaining the present set of vectors. A complete description 
of Powell's method and a detailed discussion of its efficiency and 
convergence is contained in [Ref.2]. 

An existing version of Powell's method programmed in FORTRAN for use 
on the IBM 360 Computer was used as a subroutine in the overall computer 
program written to conduct the search technique optimization. The main 
program provided the calling argument required by the subroutine to 


conduct the optimization. 


B. CURVE-FITTING TECHNIQUE 
The curve-fitting technique used was based on standard multiple 


linear regression techniques [Refs. 3 and 4] modified as follows. 





Because the test functions to be investigated were unimodal the decision 
was made to fit a quadratic surface to the experimental data. Thus, the 
order of the regression was double the number of input variables. 

An early decision required to be made concerned exactly what proced- 
ure would be used to place the test points in the input variable space. 
Two possibilities were considered. A completely random selection of the 
test points was one alternative. A second approach was to place the test 
points throughout the space in accordance with an experimental design. 
The first experimental design considered was a composite factorial design 
[Ref. 5]. This approach was attractive for objective functions of few 
input variables. However, the requirement for an excessive number of 
test points and consequently an increased number of function evaluations 
precluded the use of this design for objective functions of more than 
four variables. 

For functions of more than four variables it was decided to use an 
orthogonal design requiring 2n+] test points, where n is the number of 
input variables. The initial test point was placed in the center of the 
input variable space and the remaining points were placed at a fixed 
distance (0.5) in both the positive and negative direction along the 
co-ordinate axes passing through the center of the design. In the 
investigations a third alternative was also considered. For those cases 
where the number of observations (function evaluations) available exceeded 
the number required for the experimental design, the remaining points 
were located randomly in the input variable space. 

Once the test points were placed, the function was evaluated at these 
points. The value of input variables, the squares of the input variables 


and the value of the objective function for each test point were then 





input to the regression subroutine to determine the regression co- 
efficients of the fitted surface. The resulting fitted surface was a 


quadratic of the form: 
2 2 
a walk X, + F b. Xx. + constant term 


Provided the quadratic was positive (negative) definite an explicit 
determination of the minimum (maximum) value of the quadratic could be 
computed. In that case, the minimum (maximum) value of the quadratic 


occurs at the point: 


] i ae 
ieee 2ebP Oe: Nees 2c 


These values of xX. were computed and substituted into the true objective 
function to determine the “optimal” objective function response.. 
Preliminary results indicated that the curve-fitting technique was 
generating a large number of indefinite quadratic surfaces. This situa- 
tion occurred more commonly when random choice was used to locate the 
test points. Any indefinite quadratics generated were not useful because. 
it was not possible to find the location of the minimum response of the 
fitted surface. Thus, an "optimum" response could not be determined in 
those situations where the curve-fitting technique generated an indefinite 
quadratic. 
In an effort to overcome this problem, a procedure was introduced 
into the curve-fitting technique to reduce the number of observations 
considered by the regression procedure. The basic idea behind the pro- 
cedure was for the regression procedure to restrict its consideration 
to those test points that were close to the optimum. In that manner 


points at a distance from the optimum did not affect the fit. It was 
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felt that this procedure would insure a better fit in the area of the 
optimum and thereby, eliminate the number of indefinite quadratics 
generated. 

A search was made of the objective function values for each test 
point. The worst 25 percent of the test points were then discarded 
reducing the number of observations considered by the regression pro- 
cedure. Because the.curve-fitting procedure was designed to find the 
minimum objective response, the worst 25 percent of the test points were 
considered to be the 25 percent with the highest values of the objective 
function. This procedure was then repeated eliminating the worst 50 
percent of the test points. 

This entire procedure was programmed in FORTRAN for use on the IBM 
360 computer. The program as written is capable of handling up to 20 
input variables and 50 observations of test points. A detailed 


description of the curve-fitting program is contained in Appendix A. 


See tes! FUNCTIONS 

Two types of objective functions were considered in the investigation; 
quadratic functions, with cross product terms, and exponential functions. 
The decision to include cross product terms in the quadratic functions 
was made to preclude the possibility of achieving an exact fit when 
employing the curve-fitting technique. In the case of exponential func- 
tions, the exponent term was a quadratic function chosen so that the 
function itself was unimodal. 

The test functions used were positive definite quadratic functions. 
Thus, the optimum response was the minimum objective function value. 
The test functions were constructed so that this minimum occurred at 


values of the input variables restricted to the domain zero to one. 


1] 





This restriction limited the selection of initial starting points for 
Powell's method to points lying within this domain. Test points for the 
curve-fitting technique were also restricted to the interval zero to one 
for all input variables. 

To insure that the quadratic functions of eight variables were in 
fact positive definite, the following procedure was used to generate 
eight by eight matrices of quadratic term coefficients. An eight by 
eight matrix was generated and pre-multiplied by its transpose. The 
resulting eight by eight matrix is assured to be positive definite. 

An explicit expression of all test functions considered can be found 


in Appendix B. 


D. NUMBER OF VARIABLES 

Functions of two different levels of input variables were considered 
in the investigation. Initially, functions of three input variables 
were investigated. The investigation concluded with a study of functions 
of eight input variables. While these two cases are not all inclusive, 
it was felt they gave a comparative indication of how well the two 


techniques handled functions of low and high levels of input variables. 


E. NUMBER OF OBSERVATIONS 

Again two different levels of observations were considered. The low 
level was fixed at 2n+] observations. This was the minimum number 
required for the orthogonal design used in the curve-fitting technique. 
Twice the number of observations in the low level was arbitrarily 
decided upon for the high level. An exception to this high level number 
was made in the case of functions of three input variables. An additional 
observation was added to the high level number to allow the use of a 


composite factorial design with the curve-fitting technique. 
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F. NUMBER OF TRIALS 

Ten trials were conducted for each procedure that was non-deter- 
ministic. These procedures included all Powell Searches and those ave 
fitting techniques where random choice was used to locate some or all of 


the points. 


G. MEASURES OF EFFECTIVENESS 
Two measures of effectiveness were employed to compare results: 
1. the mean optimum response attained by optimization technique, 
2. the mean Euclidean distance between the location of the true 
optimum and the location of the optimum response found by the 


optimization technique. 





TI]. PRESENTATION OF RESULTS 


Tables I through XI contain the results of the eleven test functions 
investigated. The following definition of terms is provided to assist 
in understanding the tabulated data: 

TECHNIQUE - The name of the optimization technique used. 

POWELL - direct search technique, Powell's method of conjugate 
directions. 

C.F. DESIGN - curve-fit technique, employing experimental design 
to locate the test points. 

C.F. RANDOM - curve-fit technique, employing random choice to 
locate the test points. 

75 PERCENT C.F. DESIGN - curve-fit technique, employing experi- 
mental design to locate the test point, eliminating the 
worst 25 percent of the test points. 

50 PERCENT C.F. DESIGN - curve-fit technique, employing experi- 
mental design to locate the test points, eliminating 
the worst 50 percent of the test points. 

75 PERCENT C.F. RANDOM - curve-fit technique, employing random 
choice to locate the test points, eliminating the worst 
25 percent of the test points. 

90 PERCENT C.F. RANDOM - curve-fit technique, employing random 
choice to locate the test points, eliminating the worst 
50 percent of the test points. 

#OBS - The number of observations included in the investigation. 

OPTIMUM RESPONSE - The mean and Standard deviation, where applicable, 


of the optimum response found using the optimization technique. 
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The mean and standard deviation were computed considering only 
the results associated with positive definite quadratics gene- 
rated by the curve-fitting technique. Results associated with 
indefinite quadratic fitted surfaces were meaningless. 

DISTANCE - The mean and standard deviation, where applicable, of the 
Euclidean distance between the location of the true optimum and 
the location of the optimum response using the optimization 
technique. The mean and standard deviation were computed con- 
Sidering only the results associated with positive definite 
quadratics generated by the curve-fitting technique. Results 
associated with indefinite quadratic fitted surfaces were 
meaningless. 

REMARKS - This column contains the number of indefinite quadratic 
Surfaces Teemrere using the different curve-fitting techniques. 
In addition, curve-fitting techniques that yielded deterministic 


results are noted by the statement, 1] trial, in this column. 


is 





| 


TABLE I 
3 VARIABLES 
QUADRATIC I 
TRUE OPTIMUM = -7.9875 


OPTIMUM RESPONSE DISTANCE 

TECHNIQUE #0BS MEAN SINE ese MEAN Si DEV. REMARKS 
POWELL J.) = S566 0.526 0.38] O27 3 

ae 725/756 OF277 0.294 Oecile 
Cale 7 =) 620 -- OU -- ] trial 
DESIGN 

lO, b-=7 625 -- O70 -- ] trial 
Oar. 7 14.945] SROs | gua 1.369 7 indef 
RANDOM quad 

15 1.6504 24.01 0.608 1.026 2 inden 

quad 
pe C.F. 15 -- -- -- -- 1] trial 
DESIGN indef 
quad 

50% C.F. 15-@%=7 25255 -- 0.116 -- Tl triai 
DESIGN 
yo7oC.F. 15 = -7.2643 0.409 0.297 0.074 7 indef 
RANDOM quad 
50% C.F. 15 = =-5.879] 22509 0.554 02570 7 indef 
RANDOM quad 
NOWES: 


ie 


C.F. design technique achieved the best result. Of interest is 
the fact that increasing the number of observations did not 
improve the results. 


C.F. random technique was quite erratic. An improvement accomp- 
anied an increase in the number of observations. However, when 
eliminating the worst 25% (50%) of the test points an improve- 
ment in mean optimum response and distance was accompanied by 

an increase in the number of indefinite quadratics. 





TABLE I1 
3 VARIABLES 
QUADRATIC 2 
TRUE OPTIMUM = -59.90 


OPTIMUM RESPONSE DISTANCE 

TECHNIQUE #0BS MEAN STD DEY. MEAN SUD. DEV. REMARKS 
POWELL 7 (P2562 52 ez 0.485 O67 

159 4=56 76 Os i 0.449 0.169 
Cat. 7 -58.14 -- 0.290 -- 1] trial 
DESIGN | 

15 = -58.14 -- 0.290 -— 1] trial 
Cer. P2235 Bons 0.896 0.365 8 indef 
RANDOM quad 

15 =-59.03 WeOo7, 0.274 0.074 6 indef 

quad 

75% C.F. Ie ee 0.919 - | trial 
DESIGN 
po, C.F. P52 0 co =~ 2.929 -- Etridl 
DESIGN 
(570 .F. 15 = -39.82 -- 0.885 ~- 9 indef 
RANDOM quad 
Seo uc.F. 1s iolse3 21908 1.619 e235 8 indef 
RANDOM quad 
NOTES: 

1. Powell's method achieved the best results at both 7 and 15 
observations. 

2. At 15 observations the C.F. random techniques achieved better 
results than Powell's method. However, 6 out of 10 quadratics 
generated using this technique were indefinite. 

3. Eliminating the worst 25% (50%) of the test points did not re- 


flect any improvement using the C.F. design technique. Although 
the mean optimum response and distance improved when eliminating 
Che WoncWe 252. Of her test points using the C.F. random technique, 
the improvement was accomplished at the expense of 3 more 
indefinite quadratics being generated. 
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TABLE ITI 
3 VARIABLES 
EXPONENTIAL 1 


TRUE OPTIMUM = 4.476 x 107° 
OPTIMUM RESPONSE DISTANCE 
TECHNIQUE  #0BS MEAN STD. DEV. MEAN STD. DEV. — REMARKS 
POWELL PEG ele Oyen. On165 
G2 Beeson” OmeRae = 2 ORees Gagan 
e fe Fmesorioe. eee 0.366 oe eter 
DESIGN a 
Geol Rai eee 0.766 oe ieee 
Bre. eine ee ey a onder 
RANDOM quad 
15. 8.018x107> 2.70x107> «0.558 . 0.268 7 indef 
quad 
75% C.F. ice Wee i _- = Sal 
DESIGN “al 
quad 
50% C.F. Gus eee ee 0.389 Et 1 trial 
DESIGN 
75% C.F. 15 1056.026 1829.075 0.746 0.563 5 acer 
RANDOM quad 
50% C.F. 15 93.852x10°> 145.24x10°° 0.506 0.142 eae ndes 
RANDOM quad 
NOTES: 
1. C.F. design technique achieved the best results at 7 observa- 


tions and the 50% C.F. design technique proved to be the best 
at 15 observations. 


At 7 observations C.F. random technique achieved a better mean 
optimum response than C.F. design technique. However, 9 out of 
10 quadratics generated were indefinite. 


Eliminating the worst 25% (50%) of the test points using the 
C.F. random technique decreased the number of indefinite quad- 
ratics generated. However, the mean optimum response in both 
cases became worse. 
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TABLE IV 
3 VARIABLES 
EXPONENTIAL 2 
TRUE OPTIMUM = 0.0212 


OPTIMUM RESPONSE DISTANCE 

TECHNIQUE #0BS MEAN S DREN. MEAN STD. DEV. |= REMARKS 
POWELL 7 £00360 0.0154 0.606 0.200 

Hope 0.0265 0.0088 0.408 0.249 
Cale 7 G.0506 -~ 0.374 -- etl 
DESIGN 

1be4 20; 2545 -- 0.776 -- ] trial 
Gor. 7 0.0368 0.0092 0.694 05236 6 indef 
RANDOM quad 

15 0.0943 CO. 0652 0.641 0.149 6 indef 

quad 

PoicaG.F [Drege he 0428 -- elias -- lateral 
DESIGN 
50, C.F. fs 020350 -- 0365 -- [a eel | 
DESIGN . 
Oa. b . i 1.22x10/ 1.22x10/ 1.802 1.026 8 indef 
RANDOM quad 
SO eaC-F. 15 = 106.38 148.94 0.729 0.302 / indef 
RANDOM quad 
NOTES: 


1. C.F. design technique was best at 7 observations. Powell's 
method achieved the best results at 15 observations. 


2. C.F. random techniques generated a large number of indefinite 
quadratics. Eliminating the worst 25% (50%) of the test points 
neither improved the mean optimum response or distance, nor 
did it decrease the number of indefinite quadratics. 
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TABLE V 
3 VARIABLES 
EXPONENTIAL 3 
TRUE OPTIMUM = 0.215/ 


OPTIMUM RESPONSE DISTANCE 


indefinite quadratics. 


— TECHNIQUE #OBS MEAN Sire bey. MEAN STD. DEV. |= REMARKS 
POWELL Te 05255 OR Se Is: 0.534 05205 
15 0.2969 0.056 05336 0.110 
Gk. ieee ccoc -- G55 -- 1 trial 
DESIGN 
Loa Oe3eZ3 -- 0.464 -- 1 trial 
ae 70° SOLS D0) 2h 2 EY 7 indef 
RANDOM quad 
lo= 0.2670 0.045 O35 0.126 6 tindef 
quad 
hop C.F. 15 0.4668 -- 0.662 -- 1 trial 
DESIGN 
pe C.F. io Ol cc02 -- 0.164 -- 1 trial 
DESIGN 
Hone. F. 152 eco. 196 46.698 OL 625 Ora S 5 indef 
RANDOM quad 
2.0) Ge ae 150-3059 O2073 0330 0.054 7 indef 
RANDOM quad 
NOTES: 
1. C.F. design technique achieved the best results at 7 observa- 
tions. At 15 observations, C.F. design eliminating the worst 
50% of the test points achieved the best results. 
2. (C.F. random techniques were extremely erratic generating many 


In the case of 7 observations, one bad 
fit caused the mean optimum response to become so large as to 
be meaningless. 
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TABLE VI 
8 VARIABLES 
QUADRATIC 1 

TRUE OPTIMUM = -44.61 


OPTIMUM RESPONSE DISTANCE 
TECHNIQUE  #0BS MEAN STD.DEV. MEAN STD. DEV. 
POWELL ae ey, Teo 1.028 0.047 

34 -43.50 0.591 0.962 0.047 
BF, 17. -44.18 Be 0.644 =a 
DESIGN | 

Bante 2 7026 ales 1.87] 1.992 
Gir: 17 = = x ie 
RANDOM 

oie ode t 7.76 1.584 1.844 
75% C.F. 34-4345 1.569 0.55 0.066 
DESIGN 
50% C.F. 34. -43.14 =e 0.827 oe 
DESIGN 
75% C.F. 34 =e = ” > 
RANDOM 
50% C.F. 34 ae = ve = 
RANDOM 
NOTES: 


REMARKS 


erect 
2 indef 
quad 


10 indef 
quad 


8 indef 
quad 


7 indef 
quad 


9 indef 
quad 


10 indef 
quad 


10 indef 
quad 


1. C.F. design achieved the best results at 17 observations. At 


34 observations, Powell's method proved to be the best 
However, it 1s Significant to note that the 
results achieved using C.F. design techniques with 1/7 


technique. 


observations is better than that achieved by Powell with 34 


observations. 


2. (C.F. random techniques were a complete disaster generating 
all indefinite quadratic surfaces in 3 out of 4 investigations 


used, 
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TABLE VII 
8 VARIABLES 
QUADRATIC 2 

TRUE OPTIMUM = -9].89 


OPTIMUM RESPONSE DISTANCE 

TECHNIQUE #0BS MEAN So weve MEAN S1O..DEY . REMARKS 
POWELL 17 —=—-- 79.56 253 0.997 Ga: We 

34 -85.04 4.02 0.942 e233 
CaF. a 15.53) -- 0.763 -- 1 trial 
DESIGN 

34 -16.92 49.88 1.099 1.314 6 indef 

quad 

Cale 17 -- -- -- -- 10 indef 
RANDOM quad 

34 676.47 -- Sey -- 9 indef 

quad 
Hoe C.F . S456 707 eee sy, Saios 1.885 8 indef 
DESIGN quad 
507, C.F. 34 -- -- -- -- 10 indef 
DESIGN quad 
bon CF . 34 -- ~~ -- -- 10 indef 
RANDOM quad 
S07, 0.F . 34 -- -- -- -- 10 indef 
RANDOM quad 
NOTES: 
1. Powell's method achieved the best results at both 17 and 34 
observations. 
2. All curve-fitting techniques were inefficient. A probable 


cause of this problem was the fact that the quadratic function 
investigated here had sharply rising contours in the vicinity 
of the optimum. 
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TABLE VIII 
8 VARIABLES 
QUADRATIC 3 


TRUE OPTIMUM = -68.25 
OPTIMUM RESPONSE DISTANCE 
TECHNIQUE #0BS MEAN Sip WEY. MEAN STD. DEV. REMARKS 
POWELL VW ees 55c9 30.42 eon 0.454 
34 = -47.6/7 26.62 Pols5 0.448 
Cae 17 154.19 ~~ Oa -- Weisel! 
DESIGN 
S456 168s 3398.9 14.68 350 3 indef 
quad 
coe ae 17 -- ae Se -- 10 indef 
RANDOM quad 
34 -- -- -~ -- 10 indef 
. quad 
yon C.F . 34 =1617.68 1282.98 2.893 1.495 7 ander, 
DESIGN quad 
Oe C.F . 34 -- -- -- os 10 indef 
DESIGN quad 
eS ae 34 -- -- -- -- 10 indef 
RANDOM quad 
SUeaC. Ff. 34 -- -- -- ~- 10 indef 
RANDOM quad 
NOTES: 


F 


Powell's method achieved the best results at both 17 and 34 
observations. 


All curve-fitting techniques were inefficient. 


A probable 


cause of this problem was the fact that the quadratic function 
investigated here had sharply rising contours in the vicinity 
of the optimum. 


as 





TRUE OPTIMUM = 1.3x107” 
OPTIMUM RESPONSE DISTANCE 

TECHNIQUE  #0BS MEAN STD. DEV MEAN STD. DEV. REMARKS 
POWELL 17 1245.08x10723605.4x1072 1.072. —s0.214 

34 «8. 46x1072_-:18.38x107? 0.925 0.188 
GP. 17 Der 0.526 _- 1 trial 
DESIGN - 

ORG IGh: Meee 0.674 - 9 indef 

quad 

ofa ee Se - = ls sacs 
RANDOM quad 

34 -- -- -- -- 10 indef 

quad 

75% C.F. 34 81.93x10"” 106.26x10°” 0.813 0.038 7 indef 
DESIGN quad 
50% C.F. Jie Scene ee 0.542 = 9 indef 
DESIGN quad 
75% C.F. ye _- = - iio) ages 
RANDOM quad 
50% C.F. 34 = = eS _ 10 indef 
RANDOM quad 
NOTES: 


1. 


TABLE IX 
8 VARIABLES 
EXPONENTIAL 1 


C.F. design technique achieved the best results at 17 
observations while Powell's method proved best at 34 observa- 
tions. However, the results at 17 observations using C.F. 
design were better than those using Powell's method with 34 
observations. 


At 34 observations, C.F. design eliminating the worst 50% of 
the test points achieved results superior to Powell's method. 
However, 9 out of the 10 quadratics generated were indefinite. 


C.F. random techniques were completely inefficient qenerating 


indefinite quadratic surfaces in every situation they were 
used, 


24 






a 
"Orne. MEG sur 


SWAT? 14 
Ke” Sie gr Re? nS 


HAIN =. AOS. 
fos R- | 
les iae, GGae flee. 85! u 
ee aT 





< Rh ris Ss rf 
1U 2 a ae, Ut ge 
rf 
i ue 
. pe 
Oilers | oy} DP 
e ®, 
bE 


TECHNIQUE #0BS 
POWELL Ww 

34 
Ger. i 
DESIGN 

34 
Car, a7. 
RANDOM 

34 
oc. Fr. of 
DESIGN 
a0, C.F. 34 
DESIGN 
You O.F . 34 
RANDOM 
Succ .F ; oul 
RANDOM 
NOTES: 


1, 


TRUE OPTIMUM = 1.02x10— 


TABLE X 
8 VARIABLES 


EXPONENTIAL 2 


OPTIMUM RESPONSE 


MEAN 


61.27x10°°° 
25 


14.40x10- 


33.4x10°° 


18599x10°°° 


Zo 


94.3x10 


STD. DEV. 
108.13x10- 
23.61x10— 


6296x107 °° 


25 
a3 


25 

DISTANCE 
MEAN Ss, WEY. 
096 0.190 
0.949 0.204 
CeeeH -- 
262 0.495 
0.696 -- 


REMARKS 


1 trial 


10 indef 
quad 


10 indef 
quad 


10 indef 
quad 


7 indef 
quad 


9 indef 
quad 


10 indef 
quad 


10 indef 
quad 


At 17 observations C.F. design achieved the best results while 
Powell's method was superior at 34 observations. 


C.F. random techniques were completely inefficient generating 
indefinite quadratic surfaces in every situation they were used. 
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TABLE XI 
8 VARIABLES 
EXPONENTIAL 3 
TRUE OPTIMUM = 9.91x10° 


OPTIMUM RESPONSE DISTANCE 
TECHNIQUE  #0BS_ MEAN STD. DEV. MEAN STD. DEV. REMARKS 
POWELL WeG7ESSIOn® GOnlexigsc 1 1.073. 0.269 
SIeEOnlOn Sea l0 2 oese0| os 277 
or, ie Gexio 0.516 _- 1 trial 
DESIGN i 5 
Beso 510 1.912 1.903 4 indef 
quad 
oge 17 ae Be a _- 10 indef 
RANDOM quad 
34 1172.35x107°1147.92x10°° 1.003 ~=—- 0.349 endet 
. quad 
75% C.F. 34 21.74x107° + 12.09x10°° 0.608 0.104 Dede 
DESIGN quad 
50% C.F. 34 13.63x10.° 0.836x10°° 0.540 0.025 6 indef 
DESIGN quad 
75% C.F. 34 10° - 2.482 ae 9 indef 
RANDOM quad 
DO Cer. 34 -- -- -- -- 10 indef 
RANDOM quad 
NOES: 
1. C.F. design achieved the best results at 17 observations. 


Although Powell's method proved to be the best at 34 observations 
the results achieved were not as good as those achieved using 
C.F. design with 1/7 observations. 


At 34 observations, C.F. design eliminating the worst 25% (50%) 
of the test points improved the mean optimum response and dist- 
ance. However, when eliminating the worst 25% a decrease in the 
number of indefinite quadratics occurred, while elimination of 
the worst 50% caused an increase in the number of indefinite 
quadratics. 


C.F. random techniques were inefficient due mainly to the 
large number of indefinite quadratics generated. 
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Ieee CONCLUSIONS 


Overall, the direct search technique, Powell's method of conjugate 
directions, proved to be the best technique of those considered in the 
investigations. A particularly desirable feature of Powell's method was 
the fact that results achieved using this technique converged to the true 
optimum response when increasing the number of observations. Thus, 
Powell's method exhibited the characteristic that the more information 
(function evaluations) available, the better the results that could be 
expected to be achieved. A decrease in the distance between the loca- 
tion of the true optimum response and the location of the optimum found 
using Powell's method was also experienced as the number of observations 
was increased. 

The curve-fitting techniques, as a whole, were quite erratic. The 
biggest single problem area was the fact that the curve-fitting tech- 
niques generated a large number of indefinite quadratic surfaces. This 
Situation was prevalent in both the case where the test points were 
located by random choice and the case where experimental design was 
used to locate the test points. However, it should be noted that the 
problem of indefinite quadratics arose in the curve-fitting techniques 
employing experimental design only in those cases where the total number 
of observations exceeded that number required by the experimental design. 
In this situation, the remaining number of test points were located by 
random choice. Thus, the results achieved strongly indicate a high 
correlation between the generation of indefinite quadratic surfaces 


and the use of random choice to locate any or all of the test points. 


=> 
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A second problem encountered when using curve-fitting techniques 
was the trend toward higher standard deviations of the optimum responses. 
This situation resulted from the fact that the curve-fitting techniques 
generated many bad fits with respect to the optimum response. This 
problem coupled with a reduction in the sample size due to the genera- 
tion of indefinite quadratics led to a higher variance of optimum 
response, thus a high standard deviation. 

To overcome these problems the effect of reducing the number of 
observations considered by the curve-fitting technique by elimination 
of the worst 25% and 50% of the test points was investigated. This 
approach did not yield consistent results. In some cases the number of 
indefinite quadratics generated was reduced, while in other cases the 
number remained the same or increased. In addition, in many cases where 
the number of indefinite quadratics was reduced, an increase in the 
number of bad fits with respect to the optimum response was experienced. 

However, curve-fitting techniques do merit consideration under 
certain circumstances. Provided that the objective function did not 
have sharply rising contours in the immediate location of the true 
optimum, curve-fitting techniques employing experimental design, where 
the number of test points considered was exactly equal to the number 
required by the experimental design, provided results better than those 
achieved by Powell's method. In fact for some of the test functions 
investigated, the results achieved using the curve-fitting techniques 
employing experimental design were better than those achieved using 
Powell's method with twice the number of observations. 

Thus, on the whole Powell's method is an efficient method of 


optimization that consistently achieved better results than any of the 
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curve-fitting techniques investigated. Under certain circumstances, 
the curve-fitting technique employing experimental design to locate 
test points provided better results and was more efficient than Powell's 


method. 
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V. SUGGESTED FUTURE INVESTIGATIONS 


The problem of generating indefinite quadratic fits when using 
curve-fitting techniques was never overcome. In future investigations 
one way of addressing this problem would be to constrain the regression 
procedures so that the coefficients of the quadratic terms are al] 
positive. Another method that could be used, either in connection with 
the above technique or by itself, is to force the fitted surface 
through a certain point. As a suggestion it might prove useful to 
force the fitted surface through the best of the test points. The best 
test point being chosen as the one with the smallest function value in 
the case of a minimization problem. 

Another interesting question that could be considered is whether 
a combination of the search and curve-fitting procedures would achieve 
better results than either procedure achieved by itself. A suggested 
approach to this question would be to use Powell's search initially to 
find the general area of the optimum solution. Once this has been 
achieved, the curve-fit design procedure could be used in this general 


area to obtain the optimum solution. 
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APPENDIX A 
CURVE-FITTING PROGRAM 


The purpose of the curve-fitting program was threefold: 

1) determine the location of the test points and their respective 
functional! values, 

2) fit a quadratic surface to these test points, 

3) determine the location of the optimum response of the resulting 
"fitted" quadratic surface and compute the true functional 


value at that point. 


A. LOCATION OF TEST POINTS 

The program considered two different methods for locating test 
points in the input variable space, random choice or placement in 
accordance with an experimental design. Input data to the program pro- 
vided the basis for selection of the appropriate method. A uniform 
random number generator was used in connection with the method of 
random choice. The two experimental designs used, composite factorial 
and orthogonal, were written into the program to be constructed by the 
computer as they were needed. 

Upon selection of the appropriate method a matrix was constructed, 
each row of the matrix containing one observation of the input variables. 
At this stage the number of columns in the matrix was equal to the number 
of input variables. The size of the matrix was then expanded in the 
following manner. For each observation the value of each input variable 
was squared and stored in the corresponding row of the matrix, thus, 


doubling the number of columns of the matrix. A final colurn was added 
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to the matrix by computing the function value corresponding to each 
observation of the input variables. 

The completed matrix was then input to the curve-fitting procedure 
unless a reduction in the number of observations was called for. In 
that case the function values were scanned by the computer to determine 
the worst 25% or 50% of the values as appropriate. These worst function 
values along with their corresponding input variables and their squares 
were eliminated from the matrix reducing the number of rows in the 


matrix input to the curve-fitting procedure. 


B. FITTING THE QUADRATIC SURFACE 

Multiple linear regression techniques were used to fit a quadratic 
surface to the test points. This section of the program consisted of 
three subroutines and the procedures for computation of the regression 
coefficients. The three subroutines used were CORRE, ORDER, and MINV. 
These subroutines are included in the IBM 360 Scientific Subroutine 
Package and are stored internally on the computer. 

The purpose of subroutine CORRE was to compute the means, the 
Standard deviations and a matrix of correlation coefficients of the 
variables included in the regression. In this case, the original input 
variables, their squares and the function values. The matrix constructed 
in the initial section of the program served as input to CORRE to accomp- 
lish these tasks. 

The resulting correlation coefficient matrix from subroutine CORRE 
was then input to subroutine ORDER. The function of subroutine ORDER 
was to compute a matrix containing the correlation coefficients of the 
independent variables and a vector containing the correlation coefficients 


of the independent variables with the dependent variables. In the 
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program, the original input variables and their squares were designated 
the independent variables, and the function value was designated the 
dependent variable. 

To check the validity of the regression the matrix of correlation 
coefficients among independent variables calculated by subroutine ORDER 
was input to subroutine MINV. Subroutine MINV computed the determinant 
of this matrix. If the determinant was non-zero, the regression was 
valid and the program continued. However, if the determinant was zero 
the program terminated due to the presence of multi-collinearity in the 
regression. 

Provided the regression was valid, the next step was to compute the 
regression coefficients of the independent variables. To accomplish 
this task a portion of subroutine MULTR, also included in the IBM 360 
Scientific Subroutine Package, was used. The standard deviations com- 
puted by subroutine CORRE, and the matrix of correlation coefficients 
among the independent variables and the vector of correlation 
coefficients between the independent variables and the dependent 
variable computed by subroutine ORDER served as input to this subroutine. 
The output of the subroutine was a vector containing the regression 


coefficients. 


C. LOCATION OF OPTIMUM RESPONSE 

The vector of regression coefficients contained the necessary 
information to determine the location of the optimum response of the 
fitted surface. The length of this vector was 2n, where n is the number 


of input variables. The resulting fitted quadratic was of the form: 
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The value of the a.'s was contained in the first n elements of the vector 
of regression coefficients and the value of the b.'s was contained in 
the last n elements. 

The location of the optimum (minimum) response was determined as 


follows. The values: 


Pe Me ede Oe aes 
os = OL > | ippcas eas 


were computed. These values uniquely determined the location of the 
minimum response of the fitted quadratic provided that the quadratic 
was positive definite. A vector containing the location of the optimum 
response of the fitted surface was then substituted into the true 
objective function to determine the optimum objective response for the 


curve-fitting procedures. 
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APPENDIX B 
TEST FUNCTIONS 


IT. THREE VARIABLE FUNCTIONS 


- A. QUADRATIC 1 


eye ee lee 
Ne 8 
6 ae 


True Optimum = -/7.9875 At X, = 0.625 
Xo = 0.4 
KX. = 0./25 


B. QUADRATIC 2 


Pesylhaloq eon col ser a lesoe2 
29 34 -81.0 





45 -98 4 
True Optimum = -59.90 At x, = Q0./7 
= 0.3 
X, = 0.4 
C. EXPONENTIAL 1 
F = exp/x! [g 2 3] x+x! [-10 
eae - 6 
4 - 8 
True Optimum = 0.004476 At x, = 0.2/2/ 
X. = 0.8636 
X, = 0.3636 


5 
D. EXPONENTIAL 2 


cell pea By 


True Optimum = 0.0212 At xy = 0.6364 
Xo = 0.9318 
Ko = 0.8182 
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E. EXPONENTIAL 3 


cml peg el 


True Optimum = 0.2157 At Xy = Ue2273 
Xo = 0.8864 
X. = 0.3636 


II. EIGHT VARIABLE FUNCTIONS 


A. QUADRATIC | 


POWONLLHOANMNM 


a 


re ia oo wo ol ye {18.8 
ae Onin 0 aiene 
96 5 5 4 2? -38 .0 
2 i 24.4 
ie Ge -25, 8 
9 0 0 23.8 
12 0 24.8 
5 -10.4 
True Optimum = -44.6] At x, = 0.6 Xe =s027 
¥,=0.9 X= 0.2 
X, = 0.2 X, = 0.3 
X, = 0.8 X = 0.8 
B. QUADRATIC 2 
event ha Moy Sal eo cae say elle oles: 
103. 54-40-28 1 46 -1 75) 
50 -28 -1-20 54 -30 6 
ae Feesa 414 5 
Dail a ois dQ. 
250230. 30 ts, 
69 -45 43. 
57 -39. 
True Optimum = -91.89 At xy = 0.7 Xe = 0.6 
x = 07 X= 0.8 
ee m0 x, = 0.2 
Mae es ee One 
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QUADRATIC 3 


p= x [43 -23 16 -29 25 -22 22 
50-22 53-45 39 -39 
19 -28 24 -21 2] 
106 -69 60 -60 
76 eds AG 
4] -40 
45 
True Optimum = -68.25 At x, 7 
Xo = 
X, = 
EXPONENTIAL 1 
ever | Wo setece 02 or 100 
aes 0G 0 
4 0-2 00 0 
1004-4 1 
Bor 0 10 
Jeg | 
a Ss 
4 
True Optimum = ].3 x 10"? At x, = 
Xo 
tae 
oF 


EXPONENTIAL 2 


F = exp (1/144 x! | 806 374 473 390 456 
395 453 417 585 

587 529 696 

810 574 
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EXPONENTIAL 3 


F = exp(1/144 x! [295 171 -170 123 125 
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